{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mBcdKd8Z7WYH"
      },
      "source": [
        "<center>\n",
        "    <p style=\"text-align:center\">\n",
        "        <h1>OpenLLM</h1>\n",
        "        <img alt=\"BentoML logo\" src=\"https://raw.githubusercontent.com/bentoml/BentoML/main/docs/source/_static/img/bentoml-logo-black.png\" width=\"200\"/>\n",
        "        </br>\n",
        "        <a href=\"https://github.com/bentoml/OpenLLM\">GitHub</a>\n",
        "        |\n",
        "        <a href=\"https://l.bentoml.com/join-openllm-discord\">Community</a>\n",
        "    </p>\n",
        "</center>\n",
        "<h1 align=\"center\">Serving Llama2 with OpenLLM</h1>\n",
        "\n",
        "[OpenLLM](https://github.com/bentoml/OpenLLM) is an open-source framework for serving and operating any Large Language Models (LLMs) in production."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fqZysSucYlJv"
      },
      "source": [
        "This is a project demonstrating basic usage of OpenLLM with\n",
        "Llama 2 as an example. In this tutorial, you will learn the following:\n",
        "\n",
        "- Set up your environment to work with OpenLLM.\n",
        "- Use OpenLLM Python APIs to create a demo.\n",
        "- Serve LLMs like Llama 2 with just a single command.\n",
        "- Explore different ways to interact with the OpenLLM server.\n",
        "- Build bentos for production deployment."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZQ777iu3J69B"
      },
      "source": [
        "## Set up the environment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w22BwSTCGa44"
      },
      "source": [
        "You can try this demo in one of the following ways:\n",
        "\n",
        "1. Via Google Colab.\n",
        "\n",
        "   We recommend you run this demo on a GPU. To verify if you're using a GPU on Google Colab, check the runtime type in the top left corner.\n",
        "\n",
        "   To change the runtime type: In the toolbar menu, click **Runtime** > **Change runtime type** > Select the GPU (T4)\n",
        "   \n",
        "   Paid users may have access to more advanced GPUs (e.g, A100). For free users, the T4 GPU might occasionally be unavailable.\n",
        "\n",
        "2. (Optional) Run this project locally.\n",
        "\n",
        "    If you have a GPU, you can also run this notebook locally with:\n",
        "    \n",
        "    ```\n",
        "    git clone git@github.com:bentoml/OpenLLM.git && cd OpenLLM/examples/openllm-llama2-demo && jupyter notebook\n",
        "    ```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DZO7EF_s5kbu"
      },
      "source": [
        "### [Optional] Check GPU and memory resources"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jnDCYgl0tm0X"
      },
      "outputs": [],
      "source": [
        "##@title Check the memory and GPU info you have\n",
        "import psutil\n",
        "import torch\n",
        "\n",
        "\n",
        "ram = psutil.virtual_memory()\n",
        "ram_total = ram.total / (1024**3)\n",
        "print('MemTotal: %.2f GB' % ram_total)\n",
        "\n",
        "print('=============GPU INFO=============')\n",
        "if torch.cuda.is_available():\n",
        "  !/opt/bin/nvidia-smi || ture\n",
        "else:\n",
        "  print('GPU NOT available')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q0ZTpnVOKYbj"
      },
      "source": [
        "### Install required dependencies"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ea6PG28SsdvA",
        "outputId": "57f9b33c-b73b-4cc2-ae9d-6e0175f3cb60"
      },
      "outputs": [],
      "source": [
        "!pip install -U -q  openllm[llama,vllm] openai langchain\n",
        "!apt install tensorrt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RAVscZBnNozc"
      },
      "source": [
        "## Python API demo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8aGgALP0L4Cs"
      },
      "source": [
        "You can create a simple demo of serving LLMs quickly by using Openllm runner.\n",
        "Learn more about Runners in the [BentoML documentation](https://docs.bentoml.com/en/latest/concepts/runner.html).\n",
        "\n",
        "Firstly, let's initialize an OpenLLM Runner locally.\n",
        "- Here, we're serving the smallest Llama 2 model, the 7 billion parameter version. You can simply change 7b in the string with 13b or 70b for the larger parameter sizes. but a larger model means larger VRAM.\n",
        "- we recommend you use vLLM as the backend for better performance.\n",
        "- we set `torch_dtype` as `float16` manaually to avoid such in-compatible error that may have when loading `Bfloat16` on T4 or V100 GPUs. You do not have to do this if you are runing on more more advanced GPUs.\n",
        "  ```\n",
        "  ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5.\n",
        "  ```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "z2kiJ3Pe8teq"
      },
      "outputs": [],
      "source": [
        "import openllm\n",
        "\n",
        "\n",
        "llm = openllm.LLM('NousResearch/Nous-Hermes-llama-2-7b', backend='vllm', torch_dtype='float16', embedded=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3LOHjoBOMj2L"
      },
      "source": [
        "### Test it with an prompt  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7lUEDZkC8s-N",
        "outputId": "d310f106-8c9c-43b9-86c8-d03dba3725e4"
      },
      "outputs": [],
      "source": [
        "import asyncio\n",
        "\n",
        "import nest_asyncio\n",
        "\n",
        "\n",
        "nest_asyncio.apply()\n",
        "\n",
        "\n",
        "async def main():\n",
        "  async for gen in llm.generate_iterator('What is the weather like in San Francisco?', max_new_tokens=128):\n",
        "    print(gen.outputs[0].text, flush=True, end='')\n",
        "\n",
        "\n",
        "asyncio.run(main())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vu7Y8c4XOPiP"
      },
      "source": [
        "### Clean up\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ihFtzXWCOW8G",
        "outputId": "083803d7-9cb0-4be7-d5f8-ab1cb4d764bc"
      },
      "outputs": [],
      "source": [
        "import gc\n",
        "import torch\n",
        "\n",
        "del llm\n",
        "\n",
        "torch.cuda.empty_cache()\n",
        "gc.collect()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HTJIVFrU75zF"
      },
      "source": [
        "## OpenLLM server demo"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gI1E8Lu4Z2zi"
      },
      "source": [
        "### Launch an OpenLLM server\n",
        "\n",
        "it is quite straightforward to Launch an OpenLLM server. With just a single command\n",
        "```\n",
        "openllm start {MODEL_ID} --backend [pt|vllm|...]\n",
        "```\n",
        "\n",
        "OpenLLM supports a variety of LLMs and archtiectures. Learn more in https://github.com/bentoml/OpenLLM#-supported-models.\n",
        "\n",
        "To unblock the following steps, run it in the background via `nohup`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "1KxYxYCZ8s5D"
      },
      "outputs": [],
      "source": [
        "!nohup openllm start NousResearch/Nous-Hermes-llama-2-7b --port 8001 --dtype float16 --backend vllm > openllm.log 2>&1 &"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Cgu02pwbuOf"
      },
      "source": [
        "### [IMPORTANT] Server status check"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_57J8LBCwLza"
      },
      "source": [
        "Before you interact with the OpenLLM server, it's crucial to ensure that it is up and running. The output of the `curl` command should start with `HTTP/1.1 200 OK`, meaning everything is in order.\n",
        "\n",
        "If it says `curl: (7) Failed to connect to localhost...`, then check `./openllm.log`; likely the server has failed to start or is still in the process of starting.\n",
        "\n",
        "If it says `HTTP/1.1 503 Service Unavailable`, the server is still starting and you should wait a bit and retry."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rzVfeo9Tbytk",
        "outputId": "4d75dd4b-a248-4bc3-f675-1465dd8ea07c"
      },
      "outputs": [],
      "source": [
        "! curl -i http://127.0.0.1:8001/readyz"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jo2d1_g4-lAW"
      },
      "source": [
        "### Interact with the LLM server\n",
        "\n",
        "Use one of the following ways to access the server.\n",
        "\n",
        "1. Run the `openllm query` command to query the model:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PvWPuh6Q-6Vq",
        "outputId": "0ef77a28-f91c-4eac-fea8-3eb5724cb84b"
      },
      "outputs": [],
      "source": [
        "!openllm query --endpoint http://127.0.0.1:8001 --timeout 120 \"What is the weight of the earth?\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aBvXcOL7a__J"
      },
      "source": [
        "2. if you are in Google Colab, visit the web UI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VTvLhCDabFpe"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "\n",
        "if 'google.colab' in sys.modules:\n",
        "  # using colab proxy URL\n",
        "  from google.colab.output import eval_js\n",
        "\n",
        "  print('you are in colab runtime. please try it out in %s' % eval_js('google.colab.kernel.proxyPort(8001)'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2j25weRLUFlb"
      },
      "source": [
        "3. Use OpenLLM's built-in Python client."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AQX6pOz8BEu3",
        "outputId": "dea431a8-9bec-496d-a802-66895be41365"
      },
      "outputs": [],
      "source": [
        "import openllm\n",
        "\n",
        "\n",
        "# sync API\n",
        "client = openllm.HTTPClient('http://127.0.0.1:8001', timeout=120)\n",
        "res = client.generate('What is the weight of the earth?', max_new_tokens=8192)\n",
        "\n",
        "# Async API\n",
        "# async_client = openllm.AsyncHTTPClient(\"http://127.0.0.1:8001\", timeout=120)\n",
        "# res = await async_client.generate(\"what is the weight of the earth?\", max_new_tokens=8192)\n",
        "print(res.outputs[0].text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MNHjOpdpOpd-",
        "outputId": "738dee42-6b9b-4b72-a650-e57a7dea9f87"
      },
      "outputs": [],
      "source": [
        "# streaming\n",
        "for it in client.generate_stream('What is the weight of the earth?', max_new_tokens=2048, n=2, best_of=2):\n",
        "  #print(f'index {it.index}: {it.text} (token: {it.token_ids})')\n",
        "  print(it.text, end=\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qgq0TOZqUm_Z"
      },
      "source": [
        "4. Send a request using `curl`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wQlwT1wc_M-r",
        "outputId": "cd336122-9516-4415-b6cc-dc4f67c4f2c0"
      },
      "outputs": [],
      "source": [
        "!curl -k -X 'POST' -N \\\n",
        "  'http://127.0.0.1:8001/v1/generate_stream' \\\n",
        "  -H 'accept: text/event-stream' \\\n",
        "  -H 'Content-Type: application/json' \\\n",
        "  -d '{\"prompt\":\"write a tagline for an ice cream shop\", \"llm_config\": {\"max_new_tokens\": 256}}'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vFAYN1_o_bRS"
      },
      "source": [
        "5. Use the OpenAI compatible endpoint."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0G5clTYV_M8J",
        "outputId": "bf792612-0c21-482c-8a80-4fbc5e71ebc4"
      },
      "outputs": [],
      "source": [
        "import openai\n",
        "import os\n",
        "\n",
        "\n",
        "client = openai.OpenAI(base_url=os.getenv('OPENLLM_ENDPOINT', 'http://localhost:8001') + '/v1', api_key='na')\n",
        "models = client.models.list()\n",
        "print('Models:', models.model_dump_json(indent=2))\n",
        "model = models.data[0].id\n",
        "\n",
        "#os.environ[\"STREAM\"] = \"TRUE\"\n",
        "stream = str(os.getenv('STREAM', False)).upper() in ['TRUE', '1', 'YES', 'Y', 'ON']\n",
        "\n",
        "completions = client.completions.create(\n",
        "  prompt='Write me a tag line for an ice cream shop.', model=model, max_tokens=64, stream=stream\n",
        ")\n",
        "\n",
        "print(f'Completion result (stream={stream}):')\n",
        "if stream:\n",
        "  for chunk in completions:\n",
        "    text = chunk.choices[0].text\n",
        "    if text:\n",
        "      print(text, flush=True, end='')\n",
        "else:\n",
        "  print(completions.choices[0].text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T3JOFs9sBNyH"
      },
      "source": [
        "#### LangChain integration\n",
        "\n",
        "OpenLLM supports integration with LangChain. You can use `langchain.llms.OpenLLM` to interact with the remote OpenLLM server. You can connect to it by specifying its URL:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "GTg055FH_M5w"
      },
      "outputs": [],
      "source": [
        "from langchain.llms import OpenLLM\n",
        "\n",
        "\n",
        "llm = OpenLLM(server_url='http://localhost:8001')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Wyiz3fLoBeJ2"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import LLMChain\n",
        "from langchain.prompts import PromptTemplate\n",
        "\n",
        "\n",
        "template = 'What is a good name for a company that makes {product}?'\n",
        "\n",
        "prompt = PromptTemplate(template=template, input_variables=['product'])\n",
        "\n",
        "llm_chain = LLMChain(prompt=prompt, llm=llm)\n",
        "\n",
        "generated = llm_chain.run(product='mechanical keyboard')\n",
        "print(generated)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNZFqf-gAdgF"
      },
      "source": [
        "### Stop the server in the background"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "id": "zZX8ICQmAdTi"
      },
      "outputs": [],
      "source": [
        "!pkill -f 'bentoml|openllm'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4tUAIkMuBzPs"
      },
      "source": [
        "## Deploy Llama2 in production with BentoCloud\n",
        "\n",
        "After you test the server, you can deploy it in production using BentoCloud."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "otXwv1No1RXP"
      },
      "source": [
        "### What is BentoCloud?\n",
        "\n",
        "[BentoCloud](https://www.bentoml.com/cloud) is a fully-managed platform designed for building and operating AI applications.\n",
        "\n",
        "  * Easiest way to deploy and operate AI applications.\n",
        "  * Natively support the OpenLLM workflow and optimization.\n",
        "\n",
        "If you don't have a BentoCloud account, visit the [BentoCloud website](https://www.bentoml.com/cloud) to start a free trial.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YRtSpeyQ2A0S"
      },
      "source": [
        "You can follow the steps in these blog posts to deploy your LLM to BentoCloud:\n",
        "\n",
        "* [llama2-7b](https://www.bentoml.com/blog/deploying-llama-2-7b-on-bentocloud)\n",
        "* [llama2-13b](https://www.bentoml.com/blog/openllm-in-action-part-2-deploying-llama-2-13b-on-bentocloud)\n",
        "* [llama2-70b]()\n",
        "\n",
        "![xlarge_demo_llama_overview_bentocloud_8d10abadb0.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n-3NqbdB3PPM"
      },
      "source": [
        "### Build a Bento\n",
        "\n",
        "Use OpenLLM to build the model into a standardized distribution unit in BentoML, also known as a Bento. Command:\n",
        "\n",
        "```\n",
        "openllm build {model-id} --backend [pt|vllm]\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BKpvHxGNwhIK",
        "outputId": "30de9160-8c05-4cba-f1be-8c1fb0a6faa0"
      },
      "outputs": [],
      "source": [
        "!openllm build NousResearch/Nous-Hermes-llama-2-7b --backend vllm --dtype float16"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EDFu2jTG3n-Q"
      },
      "source": [
        "### View the Bento"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3k9816Jd3rnY",
        "outputId": "62ebf693-b293-4707-dd5d-53df15270938"
      },
      "outputs": [],
      "source": [
        "!openllm list-bentos"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i2qwy9l53uso"
      },
      "source": [
        "### Log in to BentoCloud and push the Bento\n",
        "\n",
        "To log in to BentoCloud and push the Bento to it, you need your BentoCloud endpoint URL and an API token. For more information, see [Manage access tokens](https://docs.bentoml.com/en/latest/bentocloud/how-tos/manage-access-token.html)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MTMSjC_71Xk6"
      },
      "outputs": [],
      "source": [
        "return_code = !bentoml cloud list-context\n",
        "\n",
        "if 'colab-user' not in ''.join(return_code):\n",
        "  # Log in to BentoCloud\n",
        "  endpoint = input('input endpoint (like https://xxx.cloud.bentoml.com): ')\n",
        "  token = input('input token (please follow https://docs.bentoml.com/en/latest/bentocloud/how-tos/manage-access-token.html#creating-an-api-token):')\n",
        "  !bentoml cloud login --api-token {token} --endpoint {endpoint} --context colab-user\n",
        "\n",
        "# Replace the Bento tag with your own\n",
        "!bentoml push nousresearch--nous-hermes-llama-2-7b-service:b7c3ec54b754175e006ef75696a2ba3802697078 --context colab-user"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7YwNbOF84Oer"
      },
      "source": [
        "### Create a Deployment via the BentoCloud Console\n",
        "\n",
        "Follow this [guide](https://www.bentoml.com/blog/deploying-llama-2-7b-on-bentocloud) to deploy this Bento on BentoCloud.\n",
        "\n",
        "![large_create_llama2_bento_deployment_c6f4ea390e.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QtzO813w9pR4"
      },
      "source": [
        "### Create a Deployment via the BentoML client\n",
        "\n",
        "You can find detailed configuration in [Deployment creation and update information](https://docs.bentoml.com/en/latest/bentocloud/reference/deployment-creation-and-update-info.html).\n",
        "\n",
        "📢 Make sure you have logged in to BentoCloud in the last step."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WLUH1c6rcE47"
      },
      "outputs": [],
      "source": [
        "###@title Alternatively, use the BentoML client to create a Deployment.\n",
        "import json\n",
        "\n",
        "import bentoml\n",
        "\n",
        "\n",
        "return_code = !bentoml cloud list-context\n",
        "if 'colab-user' not in ''.join(return_code):\n",
        "  print('please login first!')\n",
        "else:\n",
        "  client = bentoml.cloud.BentoCloudClient()\n",
        "  # runner config\n",
        "  runner = bentoml.cloud.Resource.for_runner(\n",
        "    resource_instance='starter-aws-g4dn-xlarge-gpu-t4-xlarge'\n",
        "    # hpa_conf={\"min_replicas\": 1, \"max_replicas\": 1},\n",
        "  )\n",
        "  # api-server hpa config\n",
        "  api_server = bentoml.cloud.Resource.for_api_server(resource_instance='starter-aws-t3-2xlarge-cpu-small')\n",
        "  hpa_conf = bentoml.cloud.Resource.for_hpa_conf(min_replicas=1, max_replicas=1)\n",
        "\n",
        "  res = client.deployment.create(\n",
        "    deployment_name='test-llama2',\n",
        "    bento='nousresearch--nous-hermes-llama-2-7b-service:b7c3ec54b754175e006ef75696a2ba3802697078',\n",
        "    context='colab-user',\n",
        "    cluster_name='default',\n",
        "    # mode=\"deployment\",\n",
        "    kube_namespace='yatai',\n",
        "    runners_config={'llm-llama-runner': runner},\n",
        "    api_server_config=api_server,\n",
        "    hpa_conf=hpa_conf,\n",
        "  )\n",
        "  print(json.dump(res, indent=4))"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
